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Abstract Body 


Problem / Background / Context: 

Researchers, policy makers, and practitioners are increasingly interested in the development of 
the core competencies associated with deeper learning, including mastery of core academic 
content, critical thinking and complex problem-solving, ability to work collaboratively, effective 
communication, learning how to learn, and the development of academic mindsets, as a way to 
better prepare all students for college and careers (Trilling, 2010). Initial research has focused on 
demonstrating the effectiveness of individual programs associated with deeper learning (Collins, 
et ah, 2013; Guha, et ah, 2014; Nichols-Barrer and Haimson, 2013). However, a recent National 
Research Council (NRC) panel identified a need for more rigorous research to fill gaps in the 
evidence base (NRC, 2012). There remains a need to determine whether deeper learning, which 
takes many forms in practice, is associated with improved educational experiences for students 
of all backgrounds. 

Purpose / Objective / Research Question / Focus of Research: 

The Study of Deeper Learning: Opportunities and Outcomes, funded by the William and Flora 
Hewlett Foundation, is a proof-of-concept study to determine whether students attending high 
schools with a mature and at least moderately well-implemented approach to promoting deeper 
learning experience greater deeper learning opportunities and outcomes than they would have 
had they not attended these schools. The study examined high schools associated with ten 
established networks from across the country that embrace the goals of deeper learning and 
promote instructional practices they believe are likely to lead to deeper learning competencies. 

To determine whether students who attend these schools experience different opportunities and 
outcomes relative to similar students who do not attend these network schools, the study 
included a sample of students from “non-network” comparison schools. In this paper, we focus 
on two central questions guiding the study: 

(1) Do students attending network schools have more opportunities to engage in deeper 
learning processes than they would have if they had not attended the network schools, and 

(2) Do individual students who experience more opportunities to engage in deeper learning 
achieve better deeper learning outcomes than students who experience fewer opportunities? 

These questions address two of the fundamental assumptions of the Hewlett Foundation’s deeper 
learning initiative: 1) that schools explicitly focused on deeper learning are indeed providing the 
opportunities they set out to provide, and do so for all students, and 2) that a relationship exists 
between an individual student’s exposure to deeper learning opportunities and key outcomes, 
regardless of participation in a network school. These assumptions are critical for an analysis of 
the outcomes of the deeper learning network schools, which we provide in paper #3. 

Improvement Initiative / Intervention / Program / Practice: 

The Study of Deeper Learning examined students’ experiences and outcomes within a sample of 
network schools that were implementing instructional approaches aligned with deeper learning. 
The networks and schools differed in their approaches to deeper learning, but some features were 
common across network schools. For example, all schools engaged students in some form of 
project-based learning that focused on the development of problem-solving skills, collaboration 
skills, and/or communication skills, while engaging students in real-world experiences. Schools 
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were engaged in authentie assessment practiees ineluding portfolios and exhibitions, and teaehers 
used frequent formative assessments to gauge student learning and inform instruetion. In 
addition, sehools provided personalized learning environments to engage students in learning and 
provide individualized support. 

Setting: 

This analysis relies on data from sehools located in six districts across two states: California and 
New York. 

Population / Participants / Subjects: 

The sample of schools for this analysis included a set of moderate- to high-implementing 
network high schools and a set of schools serving similar student populations in the same 
jurisdiction as the network school, but not belonging to one of the 10 networks. Our goal was to 
identify and recruit a “non-network” school with similar student characteristics for each network 
school so that we could conduct student-level analyses with similar student populations across 
the two groups of schools. Specific criteria for school and student sampling are included in the 
abstract for paper #1. For the analysis and findings described in this paper, we collected data 
from students within 12 network high schools, which represented 8 of the 10 networks and were 
located in five districts across two states.^ Analyses also include students attending 10 non- 
network schools located in six districts across two states. One non-network school was matched 
to two network schools, for a total of 1 1 school pairs. 

Research Design: 

This study used a quasi-experimental design to collect and analyze data from students within 
pairs of network and non-network schools operating in the same or similar district contexts, as 
described in paper #1. We used propensity score matching to identify students in the comparison 
schools who, at entry to 9* grade, shared similar characteristics (including prior achievement) to 
those in the treatment schools. For the results reported in this paper, we focused on a survey and 
assessment administered to students in 11* and 12* grades in 2012-13. This quasi-experiment 
has several limitations, outlined in paper #1 . First, as a proof-of-concept study, the study was not 
designed to be generalizable across all possible contexts. In addition, while we matched and 
weighted students on a series of measured characteristics (e.g., 8* grade incoming achievement), 
the design does not account for any unmeasured characteristics associated with students’ 
attendance at network schools. 

Data Collection and Analysis: 

The primary data sources used for the analysis in this paper include: 

• Student survey data: We administered a one-hour survey to all consented and sampled 
11* and 12* grade students in each network and non-network school in spring 2013. The 
focus of the survey was twofold: 1) to measure opportunities students experience in 
school related to the six deeper learning competencies, and 2) to measure intra- and inter- 
personal competencies and dispositions that are considered important to college and 
career readiness. For a list of the constructs measured in the survey, see Appendix Exhibit 
1 . The survey enabled us to gather data from a large representative sample of students 


^ Two of these network schools were combined for analyses since they had small student populations eligible for 
the study and were co-located on one campus. 
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across both network and non-network sehools. In total, we administered surveys to 1,762 
students from the 1 1 pairs of sehools with an average response rate of 76%. 

• OECD PISA-based Test for Schools: We administered the OECD PISA-based Test for 
Sehools to all eonsented and sampled 11* and 12* grade students in eaeh network and 
non-network sehool in spring 2013 to measure higher-order skills ineluding eomplex 
problem solving in reading, mathematies, and scienee. In total we administered the 
assessment to 1,267 students, with an average response rate of 61%^ 

• Qualitative data: We eondueted 2-day visits to all network sehools partieipating in the 
study and interviewed network leaders. These data provide illustrative examples of the 
types of opportunities for deeper learning provided to students. 

To address the first research question, we performed doubly robust ordinary least squares 
regression models that applied propensity seore weights and aeeounted for student demographies 
and pre-high sehool (i.e., middle sehool) aehievement. In addition to propensity seore weights, 
we also caleulated and applied weights that aeeounted for eonsent, sampling, and non-response. 
We performed student-level analyses separately within eaeh pair of network and non-network 
schools, and then ealculated the preeision-weighted average difference between network and 
non-network students aeross the 1 1 pairs of sehools using a fixed effeets meta-analysis. We also 
examined differential effeets by student subgroup. 

To address the seeond researeh question, we used a general three-level hierarehical linear model 
to estimate both within-sehool and between-sehool relationships between opportunity measures 
and outeomes. These models aeeounted for the nesting of students within sehools, and adjusted 
for student-level 8* grade characteristies. 

Findings / Outcomes: 

The primary findings include: 

• Students in participating network high schools reported significantly greater 
opportunities for deeper learning than students in the paired non-network schools. These 
signifieant differenees held true for all measures of opportunities for deeper learning, 
with average effect sizes ranging from 0.21 (assessments aligned with deeper learning) 
up to 0.55 (opportunities to eollaborate) (see Appendix Exhibit 2). We observed the 
largest assoeiations between network sehool attendanee and opportunities for deeper 
learning in the areas of eollaboration, interdiseiplinary learning, feedbaek to students, and 
eommunication. 

• While we focus on the average effeet of network school attendance, at the individual pair 
level, students in each of the 11 network schools reported significantly more opportunity 
for deeper learning on at least one opportunity measure than students in the paired non- 
network school. In addition, we observed no negative effects of network attendance on 
opportunities within any individual pairs. 

• We also explored whether all students in the network sehools are experieneing these 
opportunities, or whether opportunities for deeper learning are differentially experieneed 


^ For the purposes of this research we received approval to use the test with older age students than the 
international norming sample. 

^ One pair of schools was included in the survey sample but not the OECD PISA-based Test for Schools sample. 
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by subgroups of students. We foeused on three eategories: students with different levels 
of prior aehievement, gender, and grade level. Out of these categories, only gender and 
grade level revealed differences in the effects of attending a network school on 
opportunities for deeper learning. Specifically, we observed a larger positive effect of 
network school attendance on opportunities for females than males. We also found larger 
positive effects of network school attendance for 12^^ graders than graders on some 
opportunity measures, including opportunities to learn how to learn, opportunities for 
creative thinking, opportunities for real world connections, and feedback to students. 

• Finally, we examined the relationship between the opportunity for deeper learning 
measures and 1) the dispositional outeomes measured in the student survey and 2) 
students’ performanee on the OECD PISA-based Test for Sehools, regardless of network 
partieipation. Overall, we found a statistically significant correlational relationship 
among all measures of opportunities for deeper learning and all measures of 
dispositional outcomes. We also found that the measure of opportunity for complex 
problem solving had a positive, statistically significant relationship with OECD PISA- 
based Test for Schools scores, including reading, math, and science. 

In addition to these findings, the paper presented will include illustrative examples from case 
study data of the types of deeper learning opportunities experienced by students. 

Conclusions: 

The findings outlined in this paper provide a strong foundation for further exploration of the 
outcomes of deeper learning. They support several key assumptions underlying the deeper 
learning approach and demonstrate that schools can provide greater opportunities for students to 
develop the deeper learning competencies, and can do so for students from a wide range of 
ability levels. They also demonstrate a fundamental positive relationship between these 
opportunities and several outcomes, including dispositional outcomes and ability to solve 
complex problems. For further generalizability, we recommend additional studies to replicate 
these findings in additional schools and a wider range of contexts. Key take-aways include; 

• Mature, moderately well implementing deeper learning network high schools serving 
substantial populations of disadvantaged students were successful in providing greater 
opportunities for students to engage in deeper learning. Analyses of student survey data 
confirmed the first core assumption of the deeper learning initiative - that schools 
explicitly focused on deeper learning and supported by one of the ten networks offered 
greater opportunities for deeper learning. The consistent, positive findings are notable 
given the wide range of deeper learning approaches and practices in the schools. 

• The relationship between network school attendance and opportunities for deeper 
learning did not differ based on students ’ levels of prior achievement, indicating that 
opportunities were provided not just to students enrolled in more advanced courses or 
who had greater academic success in the past. 

• Opportunities experienced by individual students were associated with deeper learning 
outcomes. This relationship confirms another underlying assumption of the deeper 
learning initiative - that students who experience greater opportunities for deeper 
learning also demonstrate improved deeper learning outcomes, including complex 
problem-solving skills (as measured by the OECD PISA-based Test for Schools) and 
dispositional outcomes considered important for college and career readiness. 
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Appendix B. Tables and Figures 


Exhibit 1 : Rasch Reliabilities for eonstructs measured in the student survey 

Rasch 

reliability 


Opportunities for deeper learning 

Opportunity for Complex Problem Solving 0.9 

Opportunity to Communicate (combined) 0.83 

Opportunities to Collaborate 0.69 

Opportunity to Learn How to Learn 0.52 

Assessments Aligned with Deeper Learning 0.77 

Feedback to Students 0.75 

Interdisciplinary learning 0.78 

Opportunities for Creative Thinking 0.79 

Real World Connections 0.84 

Dispositional outcomes 

Collaborative Orientation 0.83 

Creative Orientation 0.77 

Perseverance 0.79 

Self-Management 0.81 

Academic Engagement 0.74 

Motivation to Learn 0.75 

Self-Efficacy 0.84 

Locus of Control 0.73 
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Effect Size 


Exhibit 2: Estimated Network School Effects for Opportunities to Learn 


Estimated Network School Effects: Opportunities to Learn Meta-Analysis Result 
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